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Abstract 

The Connecticut State Department of Education has established three levels of 
educational technology competencies for teachers. Level I competencies include basic 
computer skills involving the use of a microcomputer operating system as well as typical 
productivity software such as the Microsoft Office Suite. Level II competencies focus on 
teachers’ familiarity with a broad range of educational technologies as well as the 
learning theory and pedagogical issues associated with the application of those 
technologies. Level HI competencies focus on teachers’ proficiency in integrating 
technology into classroom instruction, ethical and social issues surrounding the use of 
technology, and the impact that technology has on student productivity. Over the past 
several years researchers at the University of Connecticut have developed a number of 
performance assessments to measure the extent to which teachers possess these 
educational technology competencies. The research presented herein describes the 
development and validation of an early version of the Level I Technology Assessment. It 
also describes how technology competency or accuracy scores relate to teachers’ self- 
efficacy concerning the tasks assessed by the performance measure as well as their 
interest in the use of technology. Scores obtained from the Level I performance measure 
have been found to be a valid and reliable means of assessing teacher educational 
technology competency. It has also been well received by teachers. Our research has 
also uncovered significant positive correlations between educational technology 
competency and self-efficacy, as expected. Unexpectedly, however, low correlations 
were found between interest scores and both teacher competency and self-efficacy. 
Differences were also found between the correlations of technology competency, on the 
one hand, and self-efficacy scores taken before and after the technology competency 
measure was administered. Further, the correlations are generally higher at post-test than 
at pre-test implying that teachers’ assessments of their own technology skills align more 
closely with a performance measure of these skills after taking an assessment designed to 
measure the skills 
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Introduction 

Elementary and secondary schools in the United States are spending upwards of $5 
billion per year on computer hardware and software. Despite these expenditures, 
educators and policy makers are concerned that technology is not being effectively 
integrated into our nations’ schools (International Society for Technology in Education, 
1998; Marcinkiewicz, 1996; Putnam & Borko, 2000). A primary reason for this 
disappointing state of affairs is that teachers have not been adequately prepared to use 
technology, particularly those experienced teachers who were licensed before educational 
technology use became an important part of the teacher preparation process (Marx, 
Blumenfeld, Krajcik & Soloway, 1998). In fact, Moursund and Bielefeldt (1999) 
reported that less than 50% of public school teachers in their survey are sufficiently well 
qualified in educational technology use to advise pre-service teachers assigned to them. 
Bailey and Pownell (1997) have spoken to the importance of staff development when 
attempting to integrate technology into classroom environment. Milone (1998) has even 
more strongly argued that effective professional development is vital if technology is ever 
to be used in intelligent ways with children. 

The State of Connecticut took significant steps in 1999 to improve the educational 
technology competency of its teachers by adopting a three-level model of educational 
technology competency, by providing professional development to improve the skills and 
competencies specified in that model, and by providing some support for the 
development of performance assessments to measure the extent to which the skills and 
competencies specified in the model had been acquired. The three-level model of 
educational technology competency, which is grounded in the International Society for 
Technology in Education (ISTE) (2000) standards and is similar to models developed in 
states like Colorado (Colorado Department of Education, 1999), forms a hierarchy of 
skills. Level I competencies included basic computer skills involving the use of a 
computer operating system, typical productivity software such as the Microsoft Office 
Suite, and the use of the Internet. Level II competencies focused on the teacher’s 
familiarity with a broad range of educational technologies (e.g., probes, robotics, and 
software) as well as the learning theory (e.g., constructivism and situated cognition) and 
pedagogical issues associated with the application, of technology. Level HI competencies 
focused on the teacher’s proficiency in integrating technology into classroom instruction 
and the impact it had on student productivity. This three-level model was modified in 
2001, as described below. 

The research described herein presents information on the development and validation of 
the original Level I performance assessment and the relationship between the derived 
performance scores and teachers reported self-efficacy about those same skills and their 
interest in using productivity software, educational software and other learning 
technologies. The goal of this assessment was to certify that individuals possessed a 
demonstrated proficiency with Level I technology skills and competencies prior to 
entering Level II training. 
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Methodology 

Technology Assessment 

Performance Assessment. 

Airasian (1996) classifies assessment approaches under the headings selection, supply, 
product or performance. Multiple choice, true-false and matching items are listed as 
selection procedures; completion, labeling a figure or diagram, short answer and concept 
maps are considered supply procedures; essays, research reports, portfolios, diaries and 
journals, projects and exhibits are listed as products; and dramatic performances, lab 
demonstrations, and oral presentations are considered performance measures. Early 
assessments of teacher technology competency relied heavily on the use of selection and 
supply procedures. More recently, technology assessment has included various types of 
performance measures, which, according to Airasian, are "assessments in which pupils 
carry out an activity or produce a product in order to demonstrate their knowledge and 
skill" (2001, p. 228). Performance assessments also permit those taking the assessment 
to show what they can do in a real life situation (Wiggins, 1993). As a result, 
performance assessment is sometimes also called authentic assessment. However, Nitko 
(2001) has pointed out that these two terms are not interchangeable. He suggests that 
authentic assessment "usually means presenting students with tasks that are directly 
educationally meaningful rather than indirectly meaningful" (Nitko, 2001, p. 245). He 
goes on to state that four features should be present for an assessment to be considered 
authentic: (1) the assessment should emphasize application; (2) it should focus what is 
being assessed directly, not indirectly, (3) it should present realistic problems; and (4) it 
should encourage open-ended thinking, 

Performance and authentic assessment procedures are being used increasingly in 
assessing all types of behaviors and educational outcomes. Such increased use results 
from certain advantages that performance assessment has over other assessment 
approaches, including that it is consistent with the highly regarded constructivist learning 
theory. This theory "emphasizes that students should use their previous knowledge to 
build new knowledge structures, be actively involved in exploration and inquiry through 
task-like activities, and construct meaning for themselves from educational experience 
(Nitko, 2001, p. 262). Despite its popularity, it should be noted that performance 
assessment does have disadvantages, including the complexity of developing high-quality 
tasks and scoring rubrics, the amount of time required to take and score these 
assessments, the lower reliability of such measures in comparison with more traditional 
assessment procedures, and validity concerns that are sometimes associated with the 
inability of performance assessment to effectively sample from a broad body of content. 

Description of the Assessment Tasks 

The Level I Technology Assessment described herein is a performance assessment. It 
was designed to measure an individual’s competency with the Microsoft Windows 
operating system, Microsoft Office applications of Word, Excel and PowerPoint, the 
Internet and an E-mail client. The design taxonomy included the knowledge of 
commands (knowing location and availability), comprehending how to use these 
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commands, and knowing how to organize/synthesize a sequence of commands to 
accomplish a specific task. The design also called for the assessment to be consistent 
with constructivist learning theory and be authentic (McMillan, 2001). That is, the 
assessment had to have a context that was consistent with how teachers would use the 
technology in their classrooms. In this vein the word processing tasks involved the 
writing of a memo, the spreadsheet tasks involved the construction of a spreadsheet for 
recording grades, and the presentation tasks involved the development of a lesson 
presentation. The assessment required participants to sit at a computer and complete 
specific tasks that are saved to disk and submitted for evaluation. System tasks included 
the formatting of a disk and finding files. Word processing tasks included changing the 
font, using the spell checker and thesaurus, moving text, and inserting a graphic and a 
table. Spreadsheet tasks included inserting columns and rows, changing fonts, entering 
information, calculating totals, calculating averages, and sorting data. The presentation 
tasks included inserting new slides, modifying color and background, changing fonts, 
ordering slides, organizing information, and using the spellchecker. Internet and E-mail 
tasks require participants to navigate on the World Wide Web, complete a survey and 
send an e-mail message with an attachment. 

As noted above, the form of the assessment discussed in this paper is an early version of 
the Level I Technology Assessment. An updated version of this instrument, which is 
now being used to collect data from a large sample of Connecticut teachers, assesses 
additional issues such as ethics and social concerns. Some of the tasks described above 
have also been changed. Our use of performance assessment to measure educational 
technology skills of teachers is consistent with procedures used in Idaho (Strickland, 
Salzman & Harris, 2000) and other states and with the new performance-based standards 
adopted by the National Council for Accreditation of Teacher Education (NCATE) 
(Mitchell, 2002). Information on the current Connecticut educational technology 
competencies is available at http://www.state.ct.us/sde/dsi/technology/CTTCt.pdf. 

Content Validity. The content validity of the assessment instrument was determined 
in several ways. First, the instrument was designed to be consistent with ISTE Standards 
and with the various teacher certification standards adopted by the State of Connecticut. 
Second, although the researchers were primarily responsible for instrument development, 
a variety of other individuals and experts were also involved in the conceptualization of 
the assessment and the item development process. These individuals included university 
faculty responsible for undergraduate and graduate level instruction either in technology 
or assessment, representatives from the Connecticut State Department of Education, and 
a variety of public school personnel. Third, the instrument was piloted with pre-service 
teachers at the University of Connecticut and in-service teachers involved in professional 
development programs conducted by the University. Fourth, the validity of the task 
classification system was determined by asking three content experts to determine 
whether the assessment tasks involved word processing, spreadsheet operations, 
presentation operations, Internet and E-mail operations or the computer operating system. 
All three experts agreed with the researcher's classification of all tasks. And fifth, the 
validity of the assigned scores was assessed using factor analysis, as described below. 
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Similar procedures were followed in designing the two other instruments included in this 
research. 

Task Accuracy. An accuracy score was derived for the assessment utilizing a rubric 
consisting of a series of questions answered on a dichotomous (Yes/No) scale. The 
number of questions varies by task depending on complexity. Consider, for example, the 
spreadsheet task that asks a participant to create a column, enter “Total Points” as the 
column title, and use an appropriate formula to calculate the total points obtained by each 
individual student. The accuracy rubric for this task contains four evaluation questions: 

Have the student scores been totaled? 

Are the total scores correct? 

Were correct formulae used to calculate the total scores? 

Was the Sum ( ) function used to calculate the scores? 

The entire accuracy rubric contains tasks and questions as follows: 

The operating system component contains 4 tasks evaluated with 5 questions. 

The word processing component consists of 6 tasks evaluated with 11 questions. 

The spreadsheet component consists of 7 tasks evaluated with 25 questions. 

The PowerPoint component consists of 5 tasks evaluated with 10 questions. 

The Internet and E-mail component has 3 tasks evaluated with 4 questions. 

The accuracy score for each component is calculated by adding the number of “Yes” 
responses derived from application the rubric and dividing that number by the number of 
possible points to arrive at a percentage score. Accuracy ratings are assigned by judges 
who are familiar with the technology and trained in the use of the instrument. 

Self-Efficacy Survey 

The assessment also requires participants to complete a pre-training and post-training 
confidence survey based on Bandura’s (1997) self-efficacy theory. According to 
Bandura (1989), higher levels of perceived self-efficacy are associated with higher levels 
of performance and higher task commitment when faced with tasks of increased 
difficulty. Further, although increases in self-efficacy may result from actually 
performing a task, they may also be achieved vicariously. Bandura has also shown that 
the higher one’s beliefs in one’s capabilities, the higher the goals one sets for oneself and 
the greater commitment one has to them. Self-efficacy has been the focus of a wide 
variety of studies that have demonstrated a strong link between this construct and 
behavior of various kinds, including phobias, weight loss, smoking cessation, and a 
number of others outcomes. (Bandura, 1997; Bandura, O’Leary, Taylor, Gauthier & 
Gossard, 1987; Bandura, Taylor, Williams, Mefford & Barhcas, 1985; Dzewaltowski, 
1994; Strecher et al., 1986). 

The survey used in the present research was developed by the researchers and asks 
participants to indicate the degree of confidence they have in performing specific 
educational technology tasks on a 5-point Likert scale ranging from 1 indicating very 
little confidence to 5 indicating a great deal of confidence. The items on the SE survey 
are consistent with the tasks on the performance assessment. The survey was 
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administered before (pre) and after (post) the administration of the accuracy measure in 
one sitting. 



Interest Survey 

Participants are also required to complete an interest survey that includes items 
concerning the use of productivity software, the use of educational software, and the use 
of other learning technologies. They are asked to indicate their degree of interest on a 5- 
point Likert scale ranging from 1 indicating very little interest to 5 indicating a great deal 
of interest. Again, the items on the interest survey are consistent with the tasks on the 
performance assessment. 



Administration of the Assessment 

The data reported here were obtained from 61 teachers from two large metropolitan 
districts within Connecticut. The assessment was conducted in a computer lab setting by 
the researchers and individuals trained by them in the use of the instruments. 

Participants were administered the assessment tasks in a group format of 10-15 
individuals. Each participant had his or her own computer loaded with the appropriate 
software and connected to the Internet. Participants were allowed to use the help 
functions during the assessment but they were not allowed to talk with one another. The 
assessment tasks could be completed in any order except for the pre-survey that was 
completed first and the post-survey that was completed last. A total of two hours was 
available for the assessment. 



Results 

Task Accuracy 

To support the validity of the accuracy scores, the data were factor analyzed. The 
Intemet/Email accuracy score was eliminated from further analysis because of low 
communality estimates (less than .01). The construct validity of the remaining four 
accuracy scores was investigated using Common Factor Analysis with an oblique 
rotation. A single factor, named Level I Technology Competency, emerged explaining 
56% of the common variance. Loadings ranged from .58 for the Word or word 
processing accuracy score to .93 for the Excel or spreadsheet score. Despite these high 
correlations and the single factor solution, four separate subscale scores will be used in 
the analyses that follow to enable direct comparisons with the self-efficacy and interest 
scores described below. 



Self-Efficacy 

Common Factor Analysis yielded three highly similar factors for the pre and post 
confidence surveys (See Table 1). These factors were labeled (1) Confidence Using the 
Windows Operating System and Microsoft Word, (2) Confidence using Microsoft Excel, 
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and (3) Confidence Using Microsoft PowerPoint. High relationships among the factors 
were noted with correlations ranging from .54 to .58 (See Table 2). 



Table 1 

Common Factor Analysis with Oblique Rotation: Pre-Assessment Confidence in Using 
Technology 



Factor 


Item 


Stem 


Loading 




Number 






3 


Saving files to the A drive 


.94 


Factor I 


2 


Locating files on the C drive 


.91 


Confidence using the 


1 


Formatting Disks 


.70 


Windows Operating 


6 


Using the cut and paste features within a Microsoft Word document 


.66 


System and Microsoft 




Using the spell checker in a Microsoft word document 




Word 


7 


Inserting objects into a Microsoft 


.63 




4 


Using the thesaurus in a Microsoft word document 


.62 
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.60 


Factor]! ^ 


10 


Creating formulas in a spreadsheet 


.92 


Confidence using 


11 


Using functions in a Microsoft Excel spreadsheet 


.87 


Microsoft Excel 


12 


Sorting data in a Microsoft Excel spreadsheet 


.78 




9 


Inserting rows and columns in a spreadsheet 


.64 


Factor III 


14 


Entering information onto a slide in Microsoft PowerPoint 


.97 


Confidence using 




Presentation. 




Microsoft PowerPoint 


16 


Reordering slides in a Microsoft PowerPoint presentation 


.93 




15 


Inserting new slides into a Microsoft PowerPoint 
presentation. 


.91 




18 


Modifying the color scheme and background in a Microsoft PowerPoint 
presentation 


.89 




17 


Using the spellchecker in a Microsoft PowerPoint 
Presentation. 


.77 




19 


Organizing information using subordination in a Microsoft PowerPoint 
presentation. 


.63 


Table 2 








Pre -Assessment Confidence in 


Using Technology: Factor Intercorrelation Matrix 




Confidence using the ^ ~ , T T . 

~ . 0 ^ Confidence Using Confidence using 

m Per V4 em Microsoft Excel Microsoft PowerPoint 

and Microsoft Word 


Confidence using the 
Windows Operating System 




1.00 




and Microsoft Word 






Confidence using Microsoft 
Excel 




.54 1.00 




Confidence using Microsoft 
PowerPoint 




.58 .58 1.00 
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Common Factor Analysis with an oblique rotation yielded three factors for the interest 
survey with 60% of the common variance extracted. These factors were labeled (1) 
Interest in Advanced Software, (2) Interest in Productivity Software, and (3) Interest in 
Technology Tools (See Table 3). Interco relations among the factors ranged from -.39 
(Productivity and Technology Tools) to .25 (Productivity Software and Advanced 
Software) (See Table 4). 



Table 3 

Common Factor Analysis with Oblique Rotation: Technology Interest Survey 



Factor 


Item 

Number 


Stem 


Loading 


Factor I 


7 


Using Computer Aided Design 


.79 


Interest in 


8 


Using Robotics 


.77 


advanced 


9 


Using Statistical software applications 


.77 


Software 


6 


Using PROBEWARE 


.65 




3 


Using Spreadsheets in excel 


.55 


Factor II 


4 


Using World Wide Web searches. 


.93 


Interest in 


5 


Using Internet/Email correspondence 


.74 


productivity 


1 


Performing word processing with Word. 


.65 


software 


2 


Building presentations with PowerPoint 


.58 


Factor HI 
Interest in 


13 


Using Distance Leaming/Dynacom. 


.75 


technology 


11 


Using Hypertext/hypermedia 


.51 


learning tools 


12 


Using laserdisks 


.51 



Table 4 

Technology Interest Survey: Factor Intercorrelation Matrix 





Interest in advanced 
Software 


Interest in 


Interest in 




Productivity 

Software 


Technology learning 
Tools 


Interest in advanced 
software 

Interest in Product. 


1.00 






software 
Interest in Tech. 


.25 


1.00 




learning tools 


-.39 


-.06 


1.00 




10 



Teacher Competency in Educational Technology 



10 



Reliability of Measures. 

Coefficient Alpha measures of reliability were calculated to determine the internal 
consistency of all the scales. The accuracy measure was found to have an alpha 
reliability of .78. Alphas for the three confidence scales (pre-test) were .95, .91 and .96, 
respectively. Alphas for the three interest scales were .83, .76 and .77, respectively. 

Intercorrelation of Measures 

The zero-order correlations among all the measures are shown in Table 5. As can be seen 
in that table, each of the four accuracy scores correlated significantly with each of the 
other accuracy scores with correlations ranging from .29 to .67. Such significant 
correlations should be expected given the single-factor solution derived described above. 
Like the accuracy scores, the interest scores also correlated highly among themselves 
with the correlations ranging from .34 to .66, and, as was the case for accuracy, this result 
was also not unanticipated. However, the low correlation of interest with the other 
measures was not expected. Significant correlations were also found among the three 
self-efficacy (confidence) measures at both the pre-test and post-test with the correlations 
themselves ranging from .45 to .82. Again, significant correlations were not unexpected, 
as were the generally significant correlations between accuracy scores and self-efficacy 
scores addressing the same skills. What was unexpected is the difference between the 
correlations of accuracy scores and self-efficacy as measured on the pre-test and the post- 
test. As can be seen in Table 5, the correlations are almost always higher at post-test than 
at pre-test, implying that one’s assessment of one’s own technology skills align more 
closely with a performance measure of these skills after taking an assessment designed to 
measure those skills. Interestingly, teachers in this sample also had more trouble 
responding favorably to the confidence items on the pre-test than the post-test. 
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Table 5 

Level I Technology Assessment Measures Intercorrelation Matrix 
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Discussion 

Increasing teacher use of technology is a pressing national concern that will be reduced, 
and hopefully even eliminated, only by improving the professional development of 
teachers and by developing measures that objectively verify that professional 
development has had its intended impact. The research reported herein describes the 
development of a performance assessment of teacher technology competency and the 
validation of that assessment using a variety of procedures. Because these initial 
validation results proved so encouraging, and because teachers who took the assessment 
have testified repeatedly to its quality and perceived validity, a number of districts in 
Connecticut are now using the instrument to assess whether their teachers are indeed 
technology competent. This endorsement of the instrument by several large and 
influential school districts points to the need for a device like ours as well as the 
importance that school districts are beginning to place on being able to validate that their 
teachers are technology competent. Such an instrument might also prove useful to 
schools, colleges and departments of education as they respond to the new performance- 
based criteria adopted by NCATE. 

With regard to self-efficacy, it is interesting to note that the mean scale scores for 
teachers’ perceived competency in using technology improved from pre-test to post-test, 
as did the correlation between actual performance and self-efficacy. These findings are 
consistent with those of Bandura (1997) and others who have shown that personal 
experience with a specific task can positively impact a person’s self efficacy, which in 
turn can then impact task commitment, motivation and final success. In this case, the 
personal experience through which teachers became more familiar with the task was the 
technology assessment. Although the goal of this process was to assess educational 
technology competency, we may have found that the assessment process itself positively 
impacts perceived educational technology competency. This finding is consistent with 
research on authentic assessment (Nitko, 2001) and provides encouragement to us as we 
attempt to seamlessly transition in-service teachers from educational technology training 
to assessment to classroom implementation. 
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